Join from an existing snapshot #1532
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…nto async_snapshot_generation
jumaffre
commented
Aug 26, 2020
| @@ -571,13 +588,14 @@ if(BUILD_TESTS) | |||
| NAME reconfiguration_test | |||
| PYTHON_SCRIPT ${CMAKE_SOURCE_DIR}/tests/reconfiguration.py | |||
| CONSENSUS raft | |||
| ADDITIONAL_ARGS --raft-election-timeout 4000 | |||
There was a problem hiding this comment.
Lowering the raft election timeout here (default is 100s) as the reconfiguration test now also waits for a new election to complete.
added 2 commits
August 26, 2020 12:04
jumaffre
commented
Aug 26, 2020
| auto seqno = network.tables->current_version(); | ||
| consensus->force_become_backup(seqno, sig->view); | ||
|
|
||
| reset_data(config.joining.snapshot); |
There was a problem hiding this comment.
Once the snapshot has been applied, we don't want to keep it around in enclave memory.
|
join_from_snapshot@11998 aka 20200827.8 vs master ewma over 50 builds from 11569 to 11985 |
achamayou
reviewed
Aug 27, 2020
achamayou
reviewed
Aug 27, 2020
achamayou
reviewed
Aug 27, 2020
achamayou
reviewed
Aug 27, 2020
achamayou
reviewed
Aug 27, 2020
|
|
||
| reset_data(config.joining.snapshot); | ||
| LOG_INFO_FMT( | ||
| "Joiner successfully resumed from snapshot at seqno {} and " |
There was a problem hiding this comment.
This sounds a slightly more final than it really is: we've started resuming from this snapshot, but until we see the evidence from the primary, we don't know if the snapshot is valid.
There was a problem hiding this comment.
You're right. That's the next step for this feature so I will address that in the next PR.
achamayou
reviewed
Aug 27, 2020
achamayou
reviewed
Aug 27, 2020
achamayou
approved these changes
Aug 27, 2020
agniutkarsh
approved these changes
Aug 27, 2020
Co-authored-by: Amaury Chamayou <amaury@xargs.fr>
Co-authored-by: Amaury Chamayou <amaury@xargs.fr>
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The PR that puts all the previous snapshot PRs together.
It is now possible for a node to join a network from an existing snapshot, making the catch-up phase much quicker (basically, only replaying the entries from the last snapshot to the last index).
To do so, the operator(s) of the CCF network are responsible for copying the snapshot file(s) produced by one node (under the
snapshots/folder) to the directory of the new joiner. The new joiner will automatically try to join from the latest (based on thesnapshot.idxfile name) snapshot, deserialising the snapshot in its store when joining (*).Raft will then be forced to become follower at a specific commit index. The snapshotter and the ledger will also have to be initialised at that index so that the ledger and snapshots on the joiner node are the same than those on other nodes in the network.
I've also added a new
snapshottest suite to test this more thoroughly + all the necessary changes to the Python infra to get the snapshots from one node copied over to the new node.(*) The snapshot can only be deserialised after the ledger secrets have been passed in by the network so that the snapshot can be decrypted.
Next: